Systems and methods for processing electronic images for generalized disease detection

ABSTRACT

Systems and methods are disclosed for generating a specialized machine learning model by receiving a generalized machine learning model generated by processing a plurality of first training images to predict at least one cancer characteristic, receiving a plurality of second training images, the first training images and the second training images include images of tissue specimens and/or images algorithmically generated to replicate tissue specimens, receiving a plurality of target specialized attributes related to a respective second training image of the plurality of second training images, generating a specialized machine learning model by modifying the generalized machine learning model based on the plurality of second training images and the target specialized attributes, receiving a target image corresponding to a target specimen, applying the specialized machine learning model to the target image to determine at least one characteristic of the target image, and outputting the characteristic of the target image.

RELATED APPLICATION(S)

This application claims priority to U.S. Provisional Application No.62/956,876 filed Jan. 3, 2020, the entire disclosure of which is herebyincorporated herein by reference in its entirety.

FIELD OF THE DISCLOSURE

Various embodiments of the present disclosure relate generally toimage-based specimen classification and related image processingmethods. More specifically, particular embodiments of the presentdisclosure relate to systems and methods for processing images todevelop a generalized pan-cancer machine learning model for developmentof biomarkers in clinical and pre-clinical studies.

BACKGROUND

In oncology studies, it is increasingly important to stratify differentpatient groups to develop personalized therapeutic strategies, tomeasure tumor progression, and/or to evaluate efficacy of therapies. Thecurrent practice for such stratification is to use clinical trialsamples that are relatively small compared to the needs of most machinelearning systems. For example, many Phase III clinical trials enrollfewer than 5000 patients and Phase I and Phase II clinical trials enrolleven lesser patients (e.g., Phase 1 generally enrolls less than 100patients, Phase 2 generally enrolls less than 300 patients). Using deeplearning and many other end-to-end machine learning techniques withthese small datasets is challenging due to overfitting, which results inthe model making inaccurate predictions.

Accordingly, it would be beneficial to apply machine learning technologyfor deep learning and other end-to-end machine learning techniques withsmall datasets such as those provided via clinical trials.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of thedisclosure. The background description provided herein is for thepurpose of generally presenting the context of the disclosure. Unlessotherwise indicated herein, the materials described in this section arenot prior art to the claims in this application and are not admitted tobe prior art, or suggestions of the prior art, by inclusion in thissection.

SUMMARY

According to certain aspects of the present disclosure, systems andmethods are disclosed for identifying or verifying specimen type orspecimen properties from image analysis of tissue specimens.

A method for generating a specialized machine learning model includesreceiving a generalized machine learning model generated by processing aplurality of first training images to predict at least one cancercharacteristic; receiving a plurality of second training images, whereinthe first training images and the second training images comprise imagesof tissue specimens and/or images algorithmically generated to replicatetissue specimens; receiving a plurality of target specialized attributeseach related to a respective second training image of the plurality ofsecond training images; generating a specialized machine learning modelby modifying the generalized machine learning model based on theplurality of second training images and the respective targetspecialized attributes; receiving a target image corresponding to atarget specimen; applying the specialized machine learning model to thetarget image to determine at least one characteristic of the targetimage; and outputting the at least one characteristic of the targetimage.

A system for generating a specialized machine learning model includes amemory storing instructions; and a processor executing the instructionsto perform a process including receiving a generalized machine learningmodel generated by processing a plurality of first training images topredict at least one cancer characteristic; receiving a plurality ofsecond training images, wherein the first training images and the secondtraining images comprise images of tissue specimens and/or imagesalgorithmically generated to replicate tissue specimens; receiving aplurality of target specialized attributes each related to a respectivesecond training image of the plurality of second training images;generating a specialized machine learning model by modifying thegeneralized machine learning model based on the plurality of secondtraining images and the respective target specialized attributes;receiving a target image corresponding to a target specimen; applyingthe specialized machine learning model to the target image to determineat least one characteristic of the target image; and outputting the atleast one characteristic of the target image.

A non-transitory computer-readable medium storing instructions that,when executed by processor, cause the processor to perform a method forgenerating a specialized machine learning model, the method includesreceiving a generalized machine learning model generated by processing aplurality of first training images to predict at least one cancercharacteristic; receiving a plurality of second training images, whereinthe first training images and the second training images comprise imagesof tissue specimens and/or images algorithmically generated to replicatetissue specimens; receiving a plurality of target specialized attributeseach related to a respective second training image of the plurality ofsecond training images; generating a specialized machine learning modelby modifying the generalized machine learning model based on theplurality of second training images and the respective targetspecialized attributes; receiving a target image corresponding to atarget specimen; applying the specialized machine learning model to thetarget image to determine at least one characteristic of the targetimage; and outputting the at least one characteristic of the targetimage. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory only and are not restrictive of the disclosed embodiments,as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate various exemplary embodiments andtogether with the description, serve to explain the principles of thedisclosed embodiments.

FIG. 1A illustrates an exemplary block diagram of a system and networkfor determining one or more characteristics based on pathology image(s),according to an exemplary embodiment of the present disclosure.

FIG. 1B illustrates an exemplary block diagram of a machine learningmodel, according to an exemplary embodiment of the present disclosure.

FIG. 2 is a flowchart illustrating an exemplary method for generating aspecialized machine learning model to output characteristics of targetimages, according to an exemplary embodiment of the present disclosure.

FIG. 3 illustrates an exemplary block diagram of a training module,according to an exemplary embodiment of the present disclosure.

FIG. 4 illustrates a diagram of a generalized machine learning model anda specialized machine learning model, according to an exemplaryembodiment of the present disclosure.

FIG. 5 is a flowchart of an exemplary embodiment of drug responsepredictions, according to an exemplary embodiment of the presentdisclosure.

FIG. 6 is a flowchart of an exemplary embodiment of cancer recurrencepredictions, according to an exemplary embodiment of the presentdisclosure.

FIG. 7 is a flowchart of an exemplary embodiment of drug toxicity andtissue abnormality predictions, according to an exemplary embodiment ofthe present disclosure.

FIG. 8 depicts an example system that may execute techniques presentedherein.

DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the exemplary embodiments of thepresent disclosure, examples of which are illustrated in theaccompanying drawings. Wherever possible, the same reference numberswill be used throughout the drawings to refer to the same or like parts.As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items. In the discussion that follows,relative terms such as “about,” “substantially,” “approximately,” etc.are used to indicate a possible variation of ±10% or less in a statedvalue, numeric or otherwise.

The systems, devices, and methods disclosed herein are described indetail by way of examples and with reference to the figures. Theexamples discussed herein are examples only and are provided to assistin the explanation of the apparatuses, devices, systems, and methodsdescribed herein. None of the features or components shown in thedrawings or discussed below should be taken as mandatory for anyspecific implementation of any of these devices, systems, or methodsunless specifically designated as mandatory.

Also, for any methods described, regardless of whether the method isdescribed in conjunction with a flow diagram, it should be understoodthat unless otherwise specified or required by context, any explicit orimplicit ordering of steps performed in the execution of a method doesnot imply that those steps must be performed in the order presented butinstead may be performed in a different order or in parallel.

As used herein, the term “exemplary” is used in the sense of “example,”rather than “ideal.” Moreover, the terms “a” and “an” herein do notdenote a limitation of quantity, but rather denote the presence of oneor more of the referenced items.

Pathology refers to the study of diseases. More specifically, pathologyrefers to performing tests and analysis that are used to diagnosediseases. For example, tissue samples may be placed onto slides to beviewed under a microscope by a pathologist (e.g., a physician that is anexpert at analyzing tissue samples to determine whether anyabnormalities exist). That is, pathology specimens may be cut intomultiple sections, stained, and prepared as slides for a pathologist toexamine and render a diagnosis. When uncertain of a diagnostic findingon a slide, a pathologist may order additional cut levels, stains, orother tests to gather more information from the tissue. Technician(s)may then create new slide(s) which may contain the additionalinformation for the pathologist to use in making a diagnosis. Thisprocess of creating additional slides may be time-consuming, not onlybecause it may involve retrieving the block of tissue, cutting it tomake a new a slide, and then staining the slide, but also because it maybe batched for multiple orders. This may significantly delay the finaldiagnosis that the pathologist renders. In addition, even after thedelay, there may still be no assurance that the new slide(s) will haveinformation sufficient to render a diagnosis.

Pathologists may evaluate cancer and other disease pathology slides inisolation. The present disclosure presents a consolidated workflow forimproving diagnosis of cancer and other diseases. The workflow mayintegrate, for example, slide evaluation, tasks, image analysis andcancer detection artificial intelligence (AI), annotations,consultations, and recommendations in one workstation. In particular,the present disclosure describes various exemplary AI tools that may beintegrated into the workflow to expedite and improve a pathologist'swork.

For example, computers may be used to analyze an image of a tissuesample to quickly identify whether additional information may be neededabout a particular tissue sample, and/or to highlight to a pathologistan area in which he or she should look more closely. As describedherein, this analysis may be done for specialized tasks such as clinicaltrials or for patients that potentially have a rare disease, making itharder to use AI technology to facilitate the analysis. Thus, theprocess of obtaining additional stained slides and tests may be doneautomatically before being reviewed by a pathologist. When paired withautomatic slide segmenting and staining machines and a specializedmachine learning model, this may provide a fully automated slidepreparation pipeline. This automation has, at least, the benefits of (1)minimizing an amount of time wasted by a pathologist determining thefindings of a slide by using an ineffective machine learning model(e.g., due to overcorrection), (2) minimizing the (average total) timefrom specimen acquisition to diagnosis by avoiding the additional timeconducting manual analysis or questionable machine learning analysis,(3) reducing the amount of tissue material wasted/discarded duringmanual repeated slide preparation, (4) reducing the cost of slidepreparation by partially or fully automating the procedure, (5) allowinghigher volumes of slides to be generated per tissue block such that theyare analyzed at the same time by a specialized machine learning model,contributing to more informed/precise diagnoses by reducing the overheadof requesting additional testing for a pathologist, and/or (6)identifying or verifying correct properties (e.g., pertaining to aspecimen type) of a digital pathology image, etc.

The process of using computers to assist pathologists is calledcomputational pathology. Computing methods used for computationalpathology may include, but are not limited to, statistical analysis,autonomous or machine learning, and AI. AI may include, but is notlimited to, deep learning, neural networks, classifications, clustering,and regression algorithms. By using computational pathology, lives maybe saved by helping pathologists improve their diagnostic accuracy,reliability, efficiency, and accessibility. For example, computationalpathology may be used to assist with detecting slides suspicious forcancer, thereby allowing pathologists to check and confirm their initialassessments before rendering a final diagnosis.

Histopathology refers to the study of a specimen that has been placedonto a slide. For example, a digital pathology image may be comprised ofa digitized image of a microscope slide containing the specimen (e.g., asmear). One method a pathologist may use to analyze an image on a slideis to identify nuclei and classify whether a nucleus is normal (e.g.,benign) or abnormal (e.g., malignant). To assist pathologists inidentifying and classifying nuclei, histological stains may be used tomake cells visible. Many dye-based staining systems have been developed,including periodic acid-Schiff reaction, Masson's trichrome, nissl andmethylene blue, and Haemotoxylin and Eosin (H&E). For medical diagnosis,H&E is a widely used dye-based method, with hematoxylin staining cellnuclei blue, eosin staining cytoplasm and extracellular matrix pink, andother tissue regions taking on variations of these colors. In manycases, however, H&E-stained histologic preparations do not providesufficient information for a pathologist to visually identify biomarkersthat can aid diagnosis or guide treatment. In this situation, techniquessuch as immunohistochemistry (IHC), immunofluorescence, in situhybridization (ISH), or fluorescence in situ hybridization (FISH), maybe used. IHC and immunofluorescence involve, for example, usingantibodies that bind to specific antigens in tissues enabling the visualdetection of cells expressing specific proteins of interest, which canreveal biomarkers that are not reliably identifiable to trainedpathologists based on the analysis of H&E stained slides. ISH and FISHmay be employed to assess the number of copies of genes or the abundanceof specific RNA molecules, depending on the type of probes employed(e.g. DNA probes for gene copy number and RNA probes for the assessmentof RNA expression). If these methods also fail to provide sufficientinformation to detect some biomarkers, genetic testing of the tissue maybe used to confirm if a biomarker is present (e.g., overexpression of aspecific protein or gene product in a tumor, amplification of a givengene in a cancer).

A digitized image may be prepared to show a stained microscope slide,which may allow a pathologist to manually view the image on a slide andestimate a number of stained abnormal cells in the image. However, thisprocess may be time consuming and may lead to errors in identifyingabnormalities because some abnormalities are difficult to detect.Computational processes using machine learning models and devices may beused to assist pathologists in detecting abnormalities that mayotherwise be difficult to detect. For example, AI may be used to predictbiomarkers (such as the over-expression of a protein and/or geneproduct, amplification, or mutations of specific genes) from salientregions within digital images of tissues stained using H&E and otherdye-based methods. The images of the tissues could be whole slide images(WSI), images of tissue cores within microarrays or selected areas ofinterest within a tissue section. Using staining methods like H&E, thesebiomarkers may be difficult for humans to visually detect or quantifywithout the aid of additional testing. Using AI to infer thesebiomarkers from digital images of tissues has the potential to improvepatient care, while also being faster and less expensive.

The detected biomarkers by a specialized machine learning model couldthen be used to recommend specific cancer drugs or drug combinationtherapies to be used to treat a patient, and the AI could identify whichdrugs or drug combinations are unlikely to be successful by correlatingthe detected biomarkers with a database of treatment options. This canbe used to facilitate the automatic recommendation of immunotherapydrugs to target a patient's specific cancer. Further, this could be usedfor enabling personalized cancer treatment for specific subsets ofpatients and/or rarer cancer types.

As described above, computational pathology processes and devices of thepresent disclosure may provide an integrated platform allowing a fullyautomated process including data ingestion, processing and viewing ofdigital pathology images via a web-browser or other user interface,while integrating with a laboratory information system (LIS). Further,clinical information may be aggregated using cloud-based data analysisof patient data. The data may come from hospitals, clinics, fieldresearchers, etc., and may be analyzed by machine learning, computervision, natural language processing, and/or statistical algorithms to doreal-time monitoring and forecasting of health patterns at multiplegeographic specificity levels.

The AI and machine learning techniques described above may be applied toimplementations where a limited training dataset is available. Thelimited training dataset may correspond to a small study, a clinicaltrial, and/or a rare disease such that the amount of training dataavailable is not sufficient to train a non-initiated machine learningmodel as doing so would result in overfitting and, as a result, wouldresult in the model making inaccurate predictions. According toimplementations of the disclosed subject matter, the limitations of asmall dataset may be mitigated by using a generalized machine learningmodel (e.g., a pan-cancer detection model) that is configured to learntumor characteristics, morphology, and tumor microenvironments acrosscross tissue types. The generalized machine learning model may betrained based on a plurality different cancer types and based on aplurality of different inputs including histologist, genomic inputs,radiology images, lab tests, patient characteristics, and the like, or acombination thereof. The generalized machine learning model may be usedto train a specialized machine learning model that is better suited tomake predictions for a specialized task, such as a small study, clinicaltrial, or for a rare disease, where a small set of data is available.

The generalized machine learning model may be trained based on a firstset of images and other inputs such that it is configured to receivepatient specific inputs and output a cancer characteristic. The cancercharacteristic may be a cancer diagnosis, tumor characterization,biomarker detection, or the like.

The generalized machine learning model may be optimized to generate aspecialized machine learning model, using low-shot learning techniques.The low-shot learning techniques may be used to modify the generalizedmachine learning model to develop specialized biomarkers, drug responsepredictions, and/or cancer outcome predictions for smaller datasets. Thesmaller datasets may be, for example, from small studies, clinicaltrials, or for rare diseases where it may be impossible or difficult toconduct large-scale clinical trials to collect sufficient training data.Accordingly, the disclosed subject matter leverages a generalized cancermachine learning model that uses tumor characteristics, morphology andmicroenvironment for development of biomarkers in clinical andpreclinical studies.

As further disclosed herein, digital images of pathology specimens(e.g., histology, cytology, immunohistochemistry, etc., or a combinationthereof) and any associated information (e.g., genomic, lab tests,radiology, patient characteristics, etc.) may be received and stored.Each pathology specimen may be linked to the associated information aswell as disease information about a respective disease presence, outcomestatus (response, recurrence, etc.), and/or the presence of anybiomarkers.

A generalized machine learning model may be instantiated using deeplearning and may be trained using a large amount (e.g., over 5,000, over10,000, over 100,000, over 1,000,000, etc.) of the pathology specimensthat are linked to the associated information as well the diseaseinformation. The generalized machine learning model may be trained topredict disease, biomarkers, and/or other attributes relevant to cancerdiagnosis and treatment from multiple tissue types. Based on thetraining, the generalized machine learning model may detect the presenceof cancer and/or biomarkers across a wide array of different tissuetypes such that the layers of the generalized machine learning model aretuned to identify tumor characteristics as well as normal and abnormaltissue morphology. The generalized machine learning model may be used toextract diagnostic features that can be used with a downstream machinelearning algorithm or it can be fine-tuned for new tasks.

A specialized machine learning model may be generated for applicationwith a small study (e.g., under 1000 samples, under 3,000 samples, under4,000 samples, under 5000 samples, etc.) such as a clinical trial (e.g.,phase 1, phase 2, phase 3), and/or a study for a rare disease wherelarger data samples cannot be obtained or are difficult to obtain. Thespecialized machine learning model may be generated by modifying thegeneralized machine learning model based on a specialized trainingdataset that is different than the training data set that thegeneralized machine learning model was trained on. The specializedtraining dataset may be from the small study or otherwise related to aspecialized task with small data sets. The generalized machine learningmodel may be modified to generate the specialized machine learning modelsuch that the specified machine learning model may leverage one or morelayers of the generalized machine learning model and tune or replace oneor more other layers to adapt to attributes of the small study. Morespecifically, the specialized machine learning model may leverage thecancer detection, tumor characterization, and/or biomarker detectioncapabilities of generalized machine learning model to build aspecialized model configured for the small study.

FIG. 1A illustrates a block diagram of a system and network fordetermining specimen property or image property information pertainingto digital pathology image(s), using machine learning, according to anexemplary embodiment of the present disclosure. As further disclosedherein, the system and network of FIG. 1A may be used with a generalizedmachine learning model or a specialized machine learning model.

Specifically, FIG. 1A illustrates an electronic network 120 that may beconnected to servers at hospitals, laboratories, and/or doctors'offices, etc. For example, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125, etc., may each be connected to an electronicnetwork 120, such as the Internet, through one or more computers,servers, and/or handheld mobile devices. According to an implementation,the electronic network 120 may also be connected to server systems 110,which may include processing devices that are configured to implement amachine learning model 100, in accordance with an exemplary embodimentof the disclosed subject matter.

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125may create or otherwise obtain images of one or more categories ofpathology specimens including patients' cytology specimen(s),histopathology specimen(s), slide(s) of the cytology specimen(s),histology, immunohistochemistry, digitized images of the slide(s) of thehistopathology specimen(s), or any combination thereof. The physicianservers 121, hospital servers 122, clinical trial servers 123, researchlab servers 124, and/or laboratory information systems 125 may alsoobtain any combination of patient-specific information, such as age,medical history, cancer treatment history, family history, past biopsyor cytology information, etc. The physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125 may transmit digitized slideimages and/or patient-specific information to server systems 110 overthe electronic network 120. Server system(s) 110 may include one or morestorage devices 109 for storing images and data received from at leastone of the physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. Server systems 110 may also include processing devices forprocessing images and data stored in the storage devices 109. Serversystems 110 may further include one or more machine learning tool(s) orcapabilities via the machine learning model 100. For example, theprocessing devices may include a generalized machine learning model or aspecialized machine learning model, as shown as machine learning model100, according to one embodiment. Alternatively or in addition, thepresent disclosure (or portions of the system and methods of the presentdisclosure) may be performed on a local processing device (e.g., alaptop).

The physician servers 121, hospital servers 122, clinical trial servers123, research lab servers 124, and/or laboratory information systems 125refer to systems used by pathologists for reviewing the images of theslides. In hospital settings, tissue type information may be stored in aLIS 125.

FIG. 1B illustrates an exemplary block diagram of a machine learningmodel 100 for determining specimen property or image propertyinformation pertaining to digital pathology image(s), using machinelearning.

Specifically, FIG. 1B depicts components of the machine learning model100, according to one embodiment. For example, the machine learningmodel 100 may include a specimen characterization tool 101, a dataingestion tool 102, a slide intake tool 103, a slide scanner 104, aslide manager 105, a storage 106, and a viewing application tool 108.For clarification, the machine learning model 100 shown in FIGS. 1A and1B is a previously trained and generated machine learning model (e.g., ageneralized machine learning model, specialized machine learning model,etc.). Additional disclosure is provided herein for training andgenerating different types of machine learning models that may be usedas machine learning model 100.

The specimen characterization tool 101, as described herein, refers to aprocess and system for determining a characteristic (e.g., cancercharacteristic) such as a specimen property or image property pertainingto digital pathology image(s) using a machine learning model such as thegeneralized machine learning model or the specialized machine learningmodel.

The data ingestion tool 102 refers to a process and system forfacilitating a transfer of the digital pathology images to the varioustools, modules, components, and devices of the machine learning model100 that are used for characterizing and processing the digitalpathology images, according to an exemplary embodiment.

The slide intake tool 103 refers to a process and system for scanningpathology images and converting them into a digital form, according toan exemplary embodiment. The slides may be scanned with slide scanner104, and the slide manager 105 may process the images on the slides intodigitized pathology images and store the digitized images in storage106.

The viewing application tool 108 refers to a process and system forproviding a user (e.g., pathologist) with a characterization or imageproperty information pertaining to digital pathology image(s), accordingto an exemplary embodiment. The information may be provided throughvarious output interfaces (e.g., a screen, a monitor, a storage device,and/or a web browser, etc.). As an example, the viewing application tool108 may apply an overlay layer over the digital pathology image(s) andthe overlay layer may highlight key areas of consideration. The overlaylayer may be or may be based on the output of the specimencharacterization tool 101 of the machine learning model 100.

The specimen characterization tool 101, and each of its components, maytransmit and/or receive digitized slide images and/or patientinformation to server systems 110, physician servers 121, hospitalservers 122, clinical trial servers 123, research lab servers 124,and/or laboratory information systems 125 over a network 120. Further,server systems 110 may include storage devices for storing images anddata received from at least one of the Specimen characterization tool101, the data ingestion tool 102, the slide intake tool 103, the slidescanner 104, the slide manager 105, and viewing application tool 108.Server systems 110 may also include processing devices for processingimages and data stored in the storage devices. Server systems 110 mayfurther include one or more machine learning tool(s) or capabilities,e.g., due to the processing devices. Alternatively or in addition, thepresent disclosure (or portions of the system and methods of the presentdisclosure) may be performed on a local processing device (e.g., alaptop).

The specimen characterization tool 101 may provide the output of themachine learning model 100 (e.g., a generalized machine learning model,a specialized machine learning model, etc.). As an example, the slideintake tool 103 and the data ingestion tool 102 may receive inputs tothe generalized machine learning model or a specialized machine learningmodel and the specimen characterization tool may identify biomarkers inthe slides based on the data, and output an image highlighting thebiomarkers via the viewing application tool 108.

Any of the above devices, tools, and modules may be located on a devicethat may be connected to an electronic network 120, such as the Internetor a cloud service provider, through one or more computers, servers,and/or handheld mobile devices.

FIG. 2 shows a flowchart 200 for outputting at least one characteristicof a specialized target image, in accordance with exemplaryimplementations of the disclosed subject matter. At 202 of FIG. 2, ageneralized machine learning model may be generated. The generalizedmachine learning model may be generated to predict at least one cancercharacteristic such as a diagnosis, a tissue characterization, abiomarker detection or the like. The generalized machine learning modelmay make predictions (e.g., biomarker detection) for different cancertypes based on images of tissue specimens such as human tissue, animaltissue, or any applicable tissue and/or images that are algorithmicallygenerated to replicate human tissue, animal tissue or any otherapplicable tissue. Tissue specimens may be from a single tissue specimenor multiple tissue specimen. At 204, the generalized machine learningmodel may be received, determined, and/or located at a training modulesuch as training module 300 of FIG. 3, as further disclosed herein. At206, a plurality of specialized training images of human tissue, animaltissue, or any applicable tissue and/or images that are algorithmicallygenerated to replicate human tissue, animal tissue, or any applicabletissue, may be received. The plurality of specialized training imagesmay correspond to a small study (e.g., clinical trial, rare disease,etc.) where only a limited amount of data is available. The specializedtraining images may all correspond to the same category of pathologyspecimens, as disclosed herein. At 208, a plurality of targetspecialized attributes each related to a respective specialized trainingimage may be received. The attributes may be related to the respectivepatients based on whom the specialized training images are generated,may be based on the respective procedures, respective treatments, and/orother respective attributes. At 210, a specialized machine learningmodel may be generated by modifying the generalized machine learningmodel based on the plurality of specialized images received at 206 andthe target specialized attributes received at 208. The specializedmachine learning model generated at 210 may correspond to the machinelearning model 100 of FIG. 1A.

A target image to be analyzed using the specialized machine learningmodel is received at 212. The target image may correspond to an image tobe analyzed based on the specialized training dataset represented by theplurality of specialized training images received at 206. At 214, thespecialized machine learning model may be applied to the target image todetermine at least one characteristic of the target image. The at leastone characteristic of the target image may be outputted via, forexample, a report, a display, or any other applicable output, as furtherdiscussed herein.

The generalized machine learning model generated at 202 of FIG. 2 may bean end-to-end machine learning module, which may be instantiated usingdeep learning. The generalized machine learning model may detect thepresence or absence of cancer across more than one tissue type (e.g.,prostate cancer, breast cancer, bladder cancer, etc.). It may alsodetect additional biomarkers or information important for staging. Forexample, for bladder cancer, the generalized machine learning model mayoutput the presence or absence of muscularis propria, a muscle thatneeds to be detected for bladder cancer staging. The generalized machinelearning model may be trained with large amounts of data to predictdisease, biomarkers, and other attributes relevant to cancer treatmentfrom multiple tissue types. Through this process, it may detect thepresence of cancer and/or biomarkers across a wide array of differenttissue types such that its layers are built upon an understanding oftumor characteristics as well as normal and abnormal tissue morphology.The generalized machine learning model may be used to extract diagnosticfeatures that can be used with a downstream machine learning algorithmor it can be “fine-tuned” for new tasks, as further disclosed herein.

To generate the generalized machine learning model at 202, a patientdataset including a large plurality of digital images of pathologyspecimens (e.g., histology, cytology, immunohistochemistry, etc.) may bereceived. The pathology specimens may be digital images generated basedon physical biopsy samples, as disclosed herein, or may be images thatare algorithmically generated to replicate human tissue, animal tissue,or any applicable tissue, by, for example, a rendering system or agenerative adversarial model. Patient associated information (genomicinformation, lab tests, radiology, patient characteristics, patientinformation, treatment information, etc.) may also be received as partof the patient dataset. Additionally, as part of training the machinelearning model, each patient dataset may be paired with information orindications about a cancer characteristic outputs (e.g., biomarkers)such as disease presence/absence, presence of staging variables (e.g.,muscularis propria for bladder cancer), classification of the form ofcancer (e.g., lobular or ductal for breast cancer), and other relevantvariables for different cancer types, outcome status (e.g., response,recurrence, etc.) and/or the presence of any biomarkers.

The patient dataset, patient associated information, and/or the cancercharacteristic outputs may be received from any one or any combinationof the server systems 110, physician servers 121, hospital servers 122,clinical trial servers 123, research lab servers 124, and/or laboratoryinformation systems 125. Images used for training may come from realsources (e.g., humans, animals, etc.) or may come from synthetic sources(e.g., graphics rendering engines, 3D models, etc.). Examples of digitalpathology images may include (a) digitized slides stained with a varietyof stains, such as (but not limited to) H&E, Hematoxylin alone, IHC,molecular pathology, etc.; and/or (b) digitized tissue samples from a 3Dimaging device, such as microCT.

The generalized machine learning model may be generated based onapplying the patient dataset and the patient associated informationpaired with the cancer characteristic output to a machine learningalgorithm. The machine learning algorithm may accept, as inputs, thepathology specimens, the patient associated information, and the cancercharacteristic outputs and implement training using one or moretechniques. For example, the generalized machine learning model may betrained in one or more Convolutional Neural Networks (CNN), CNN withmultiple-instance learning or multi-label multiple instance learning,Recurrent Neural Networks (RNN), Long-short term memory RNN (LSTM),Gated Recurrent Unit RNN (GRU), graph convolution networks, or the likeor a combination thereof. Convolutional neural networks can directlylearn the image feature representations necessary for discriminatingamong characteristics, which can work extremely well when there arelarge amounts of data to train on for each specimen, whereas the othermethods can be used with either traditional computer vision features,e.g., SURF or SIFT, or with learned embeddings (e.g., descriptors)produced by a trained convolutional neural network, which can yieldadvantages when there are only small amounts of data to train on. Thetrained machine learning model may be configured to provide cancercharacteristics as outputs based on patient data and patient associatedinformation.

The generalized machine learning model may receive a patient dataset(e.g., one or more digital images of pathology specimen (e.g.,histology, cytology, immunohistochemistry etc.)) as well as patientassociated information (genomic, lab tests, radiology, patientcharacteristics etc.). The generalized machine learning model's trainedalgorithm may be applied to the patient dataset and the patientassociated information to determine one or more cancer characteristicssuch as one or more regions of cancer in the digital images. The cancercharacteristics may not be cancer specific such that the generalizedmachine learning model may provide cancer characteristics across cancertypes, if any. The cancer characteristics may be spatially varyingacross one or more digital slides.

The output of the generalized machine learning model (i.e., the one ormore cancer characteristics, if any) may be provided to a storagecomponent (e.g., cloud storage, hard drive, network drive, etc.). If aspatially varying determination is made, the corresponding cancercharacteristic(s) may be provided for digital display as for example,coordinates, bitmasks, overlays, or the like or a combination thereof.

FIG. 3 shows an example training module 300 to train either thegeneralized machine learning model or a specialized machine learningmodel, as further disclosed herein. As shown in FIG. 3, training data302 may include one or more of pathology images 304 (e.g., digitalrepresentation of biopsied images), patient data 306 (e.g., a patientdataset), and known outcomes 308 (e.g., cancer characteristics) relatedto the patient data 306. The training data 302 and a training algorithm310 may be provided to a training component 320 that may apply thetraining data 302 to the training algorithm 310 in order to generate amachine learning model.

At 206 of FIG. 2, a plurality of target specialized training images ofhuman tissue, animal tissue, or any applicable tissue and/or images thatare algorithmically generated to replicate human tissue, animal tissue,or any applicable tissue may be provided. The target specializedtraining images may correspond to images that are generated in a smallstudy and may be directed to a specific cancer based implementation. Thepathology specimens may be digital images generated based on physicalbiopsy samples, as disclosed herein, or may be images that arealgorithmically generated to replicate human tissue, animal tissue, orany applicable tissue by, for example, a rendering system or agenerative adversarial model.

The target specialized training images for a target specialized task(e.g., corresponding to a rare disease, a small study, a clinical study,etc.) may be received from any one or any combination of the serversystems 110, physician servers 121, hospital servers 122, clinical trialservers 123, research lab servers 124, and/or laboratory informationsystems 125. Images used for training may come from real sources (e.g.,humans, animals, etc.) or may come from synthetic sources (e.g.,graphics rendering engines, 3D models, etc.). Examples of such digitalpathology images may include (a) digitized slides stained with a varietyof stains, such as (but not limited to) H&E, Hematoxylin alone, IHC,molecular pathology, etc.; and/or (b) digitized tissue samples from a 3Dimaging device, such as microCT.

Compared to the images received as part of the training dataset of thegeneralized machine learning model, the number of target specializedtraining images for training a specialized machine learning model may besubstantially lower (e.g., by one or two magnitudes). The lower numberof target specialized training images may be a result of the targetspecialized training images corresponding to the target specialized taskfor a small study, clinical study, or a rare disease where an largernumber of training data is not available.

At 208, a plurality of target specialized attributes related to arespective specialized training image may be received. The targetspecialized attributes may be paired with the training image and mayinclude patient associated information (genomic information, lab tests,radiology, patient characteristics, patient information, treatmentinformation, etc.). Additionally, the target specialized attributes mayinclude information or indications about a cancer characteristic outputs(e.g., biomarkers) such as disease presence/absence, presence of stagingvariables, drug response, toxicity, classification of the form ofcancer, and other relevant variables for different cancer types, outcomestatus and/or the presence of any biomarkers.

At 210 of FIG. 2, a specialized machine learning model may be generatedfor a target specialized task. The specialized machine learning modelmay be generated by modifying the generalized machine learning model byfirst modifying the generalized machine learning model to have theappropriate outputs for the target specialized task. A generalizedmachine learning model may be trained utilizing data associated withvarious cancer types and other output targets (e.g., severity of cancer,mutations present, etc.). The generalized machine learning model may beable to recognize tumor regions/characteristics for different types ofcancer types without specifically providing a cancer type that acorresponding tissue is associated with. Such an ability to recognizeregions/characteristics for different types of cancer provides thegeneralized machine learning model with internal representations (e.g.,parameters, layers, weights associated with layers, relationships, etc.)that work effectively for other tasks where there is less data.According an implementation, a biomarker detection system (e.g.,specialized machine learning model) is initialized with the generalizedmachine learning model's parameters and, for example, the output layer,if implemented in a form of neural network, is re-initialized to befine-tuned to infer the biomarker task. The fine-tuning training canthen be done with gradient descent. Optionally, this process can beconstrained by only training the last M layers of the network or usingmethods such as L2-SP to limit the ability for the network to overfit.Additionally, specialized machine learning model may be generated bymodifying the generalized machine learning model using the plurality ofspecialized training images of 206 and the plurality of targetspecialized attributes of 208. The generalized machine learning modelmay be modified to have the appropriate outputs for the biomarkerdetection task. Additionally or alternatively, the generalized machinelearning model may be modified to extract features from samples for usewith the specialized machine learning model.

The specialized machine learning model may be generated using a smallamount of data by modifying the generalized machine learning model byfine-tuning (e.g., re-training) one or more layers of the generalizedmachine learning model using the specialized task and related material(e.g., specialized training images, target specialized attributes,etc.). The fine-tuning may be conducted using L2-SP, Deep LearningTransfer (DELTA) (e.g., using a feature map), and/or one or more otherapproaches designed to improve generalization. Alternatively or inaddition, the specialized machine learning model may be generated usinglarge-margin methods built on top of the generalized machine learningmodel to improve generalization. Alternatively or in addition, thespecialized machine learning model may be generated using methods forlow-shot learning. Alternatively or in addition, the specialized machinelearning model may be generated using the generalized machine learningmodel to extract features and then training a model based on thosefeatures (e.g., nearest neighbor, random forest, support vector machine,neural network, etc.).

The specialized machine learning model may be generated by performingtransfer learning in deep learning using the generalized machinelearning model. Transfer learning may be used to accelerate the trainingof the specialized machine learning model as either a weightinitialization scheme or feature extraction method. The weights of thegeneralize machine learning model pre-trained by the training datasetwith a sufficiently large number of instances may provide a betterinitialization for the target specialized task based specialized machinelearning model, than a random initializations.

According to a weight initialization scheme, the weights in lowerconvolution layers may be fixed and weights in upper layers mayre-trained using data from the target task and its related material(e.g., specialized training images, target specialized attributes,etc.). The weights in re-used layers may be used as the starting pointfor the training process and adapted in response to the target task.This weight initialization scheme may treat transfer learning as a typeof weight initialization scheme.

Alternately, in accordance with a feature extraction scheme, the weightsof the generalized machine learning network may not be adapted whentraining the specialized machine learning network, in response to thetarget task, such that only new layers after the reused layers may betrained to interpret their output.

Accordingly, the generalized machine learning model and the specializedmachine learning model may share one or more layers and may have atleast one layer that is different than each other. As an example, theoutput layer of the generalized machine learning model may be modifiedat 210 of FIG. 2 such that a target image received as an input at thespecialized machine learning model provides a different result than ifthe same target image was received as an input at the generalizedmachine learning model.

At 210 of FIG. 2, the specialized machine learning model may be trainedusing a training module 300 of FIG. 3 in a manner similar to thatdescribed herein for training the generalized machine learning model. Asshown in FIG. 3, training data 302 may include one or more of pathologyimages 304 (e.g., digital representation of biopsied images), patientdata 306 (e.g., a patient dataset), and known outcomes 308 (e.g., cancercharacteristics) related to the patient data 306. The pathology images304 may include the specialized training images of 206 of FIG. 2. Theknown outcomes 308 may include the target specialized attributes of 208of FIG. 2. The training data 302 and a training algorithm 310 may beprovided to a training component 320 that may apply the training data302 to the training algorithm 310 in order to generate a specializedmachine learning model.

FIG. 4 is a diagram that shows a generalized machine learning model 400and a specialized machine learning model 420. The generalized machinelearning model may have a number of inner layers 402 as well as a firstouter layer 404 and a second outer layer 406. According to an example,an outer layer may be a layer that is formed later in the training of amachine learning model in comparison to an inner layer. According toanother example, an outer layer may be more specific compared to aninner layer that is more general. The generalized machine learning model400 may be generated using a large amount of training data to outputcancer characteristics across different cancer types, as disclosedherein. The generalized machine learning model 400 may be provided totraining module 300 along with the specialized training images of 206and target specialized attributes 208 of FIG. 2.

The training module 300 may be configured to generate the specializedmachine learning model 420 by maintaining the inner layers 402 of thegeneralized machine learning model and modifying the first outer layer404 and second outer layer 406 to a first outer layer 424 and secondouter layer 426. Training of the specialized machine learning model 420may be initialized based on the inner layers 402 and the training module300 may replace, modify, or tweak the first outer layer 404 and secondouter layer 406 based on the specialized training images of 206 andtarget specialized attributes 208 of FIG. 2. Accordingly, thespecialized machine learning model 420 may be trained using a relativelysmall amount of data and may leverage the previously trained innerlayers 402 of the generalized machine learning model 400. As thegeneralized machine learning model 400 is trained to identify cancercharacteristics, the inner layers 402 may be provide a more applicableinitialization for the specialized machine learning model 420 thaninitializing the specialized machine learning model 420 without theinner layers 402.

It will be understood that although first outer layer 404 and secondouter layer 406 are shown the be modified, the any number of layers lessthan the total number of layers in the generalized machine learningmodel 400 may be modified to generate the specialized machine learningmodel 420. As an example, the second outer layer 406 may be an outputlayer and only the output layer of the generalized machine learningmodel may be modified when generating the specialized machine learningmodel 420. Additionally, it will be understood that although the innerlayers 402 of the generalized machine learning model 400 are maintainedwhen training the specialized machine learning model 420,implementations of the disclosed subject matter are not limited to innerlayers. Any applicable layers of the generalized machine learning model400 may be maintained or modified/replaced to generate the specializedmachine learning model 420.

The specialized machine learning model may be used to make predictionssuch as to determine one or more biomarkers across cancer types. Thespecialized machine learning model may determine the presence or absenceof one or more biomarkers in one or more slide images. Thisdetermination may be spatially varying across a target image (e.g., adigital pathology slide) such that different tumors in different regionsof the slides are determined to have the presence or absence ofdifferent biomarkers.

The machine learning model generated at 210 may be the same as orsimilar to the machine learning model 100 of FIG. 1A and may receivetarget images, and patient information from one or more of the physicianservers 121, hospital servers 122, clinical trial servers 123, researchlab servers 124, and/or laboratory information systems 125, etc. At 212,the target image corresponding to a target specimen may be received. At214, the specialized machine learning model may be applied to the targetimage to determine at least one characteristic of the target image. Theat least one characteristic may be a cancer characteristic associatedwith the target specialized task based on which the specialized machinelearning model was generated. At 216, the at least one characteristic ofthe target image may be output via one or more output interfaces (e.g.,a screen, a monitor, a storage device, and/or a web browser, etc.). Theoutput characteristic may be a specimen type (e.g., cancer prediction,drug response, cancer reoccurrence rate, toxicity, tissue abnormality,etc.). Accordingly, the output at 216 may be the prediction of thespecimen type based on the target image received at 212. As an example,the viewing application tool 108 of FIG. 1A may apply an overlay layerover the digital pathology image(s) and the overlay layer may highlightkey areas of consideration. The output may be provided as coordinates,bitmasks, overlays, or the like or a combination thereof.

A specialized machine learning model may be used for a number ofimplementations such as, not limited to, drug response predictions forpatient stratification in clinical trials, cancer recurrencepredictions, drug toxicity or abnormality predictions, or the like.

FIG. 5 shows an example implementation of a specialized machine learningmodel, generated using a generalized machine model, for drug responsepredictions. Developing biomarkers for trial drugs is traditionallyconducted through clinical studies where the sample size is typicallyunder 5000 patients. With such small datasets, it is difficult to fullyunderstand the underlying disease mechanism and to predict patientcharacteristics for a treatment. Techniques disclosed herein including ageneralized machine learning model that fully characterizes tumors theirmorphology can be used as an initialization step for detectingbiomarkers for identifying which patients will respond to a treatment inclinical trials, and what the response may be. As shown in FIG. 5, at502, a generalized machine learning model may be received at 502. Thegeneralized machine learning model may be adjusted to have drug responseprediction outputs at 504. The adjustment may be made by adjusting theweights in one or more layers of the generalized machine learning modeland/or the weights of the output layer of the generalized machinelearning model, and/or modifying attributes of the outputs of thegeneralized machine learning model.

The adjusted machine learning model may be provided to training module300. The training module 300 may be configured to generate thespecialized machine learning model, at 510, by maintaining the one ormore layers (e.g., inner layers) of the generalized machine learningmodel and modifying one or more layers (e.g., the outer layers) of themachine learning model. The specialized machine learning model may betrained using a relatively small amount of data and may leverage thepreviously trained layers of the generalized machine learning model. Thespecialized machine learning model may be trained by providing pathologyimages and corresponding patient data for patients that were provided atarget drug. Additionally, known outcomes of the target drug may also beprovided to the training module 300 to train the specialized machinelearning model.

At 510, the specialized machine learning model may be generated based onmodifying the generalized machine learning model received at 520 andtraining based on specialized training images from patients that areprovided the target drug, as well as their known responses to the drug.The specialized machine learning model generated at 510 may be used topredict drug response outcomes based on one more target images. FIG. 5shows steps 212, 214, and 216 of FIG. 2 and disclosure related to thesesteps is not repeated here for brevity. At 212, a target imagecorresponding to a target specimen may be received. At 214, thespecialized machine learning model generated at 510 may be applied tothe target image to determine a characteristic of the target image. Inthe implementation provided in FIG. 5, the characteristic of the targetimage may be the response (e.g., positive, negative, neutral, predictissues, etc.) that the patient from whom the target image was capturedmay have to the target drug. At 216, the characteristic may be output inaccordance with the disclosure provided herein.

According to an implementation, the generalized machine learning modelreceived in the example implementations provided in FIG. 5 (i.e., at502), FIG. 6 (i.e., at 602), and FIG. 7 (i.e., at 702), may be the samegeneralized machine learning model. Each of the respective specializedmachine learning models generated at 510, 610, and 710 may beinitialized using all or some of the layers from the generalized machinelearning model. However, each of the respective specialized machinelearning models generated at 510, 610, and 710 may be different suchthat they tuned to each of their specific specialized tasks.Accordingly, a given input image provided to each of the specializedmachine learning models generated at 510, 610, and 710 may result indifferent outputs, based on the differences between the specializedmachine learning models generated at 510, 610, and 710.

FIG. 6 shows an example implementation of a specialized machine learningmodel, generated using a generalized machine model, for cancerrecurrence predictions. Recurrence of cancer may occur when cancerreoccurs after treatment (e.g., a successful or unsuccessful treatment).Knowing whether a cancer will recur may enable better treatmentplanning. For example, knowing a potential recurrence probability basedon one or more given treatments (e.g., Immunotherapy, Chimeric antigenreceptor T (CART-T) cell-based therapy, etc.), may enable customizationor tailoring of the treatments for each patient. Additionally, newertreatment mechanisms may affect tumor's recurrence in later stages andknowing a probability associated with such late stage recurrences mayhelp mitigate the recurrence rates. However, building a machine learningmodel to assess cancer recurrence directly from traditional studies ischallenging due to limited number of datasets. Techniques disclosedherein including a generalized machine learning model that characterizestumors their morphology can be used for initiating a specialized machinelearning model for predicting cancer recurrence in studies with limiteddatasets. As shown in FIG. 6, at 602, a generalized machine learningmodel may be received at 602. The generalized machine learning model maybe adjusted to have cancer recurrence prediction outputs at 604. Theadjustment may be made by adjusting the weights in one or more layers ofthe generalized machine learning model and/or the weights of the outputlayer of the generalized machine learning model, and/or modifyingattributes of the outputs of the generalized machine learning model.

The adjusted machine learning model may be provided to training module300. The training module 300 may be configured to generate thespecialized machine learning model, at 610, by maintaining the one ormore layers (e.g., inner layers) of the generalized machine learningmodel and modifying one or more layers (e.g., the outer layers) of themachine learning model. The specialized machine learning model may betrained using a relatively small amount of data and may leverage thepreviously trained layers of the generalized machine learning model. Thespecialized machine learning model may be trained by providing pathologyimages and corresponding patient data for patients that either exhibitedcancer recurrence or did not exhibit cancer recurrence. Additionally,the known outcomes of cancer recurrence or lack of recurrence may alsobe provided to the training module 300 to train the specialized machinelearning model.

At 610, the specialized machine learning model may be generated based onmodifying the generalized machine learning model received at 620 andtraining based on specialized training images from patients thatexhibited or did not exhibit cancer recurrence, as well as their knownresponses to the drug. The specialized machine learning model generatedat 610 may be used to predict cancer recurrence outcomes based on onemore target images. FIG. 6 shows steps 212, 214, and 216 of FIG. 2 anddisclosure related to these steps is not repeated here for brevity. At212, a target image corresponding to a target specimen may be received.At 214, the specialized machine learning model generated at 610 may beapplied to the target image to determine a characteristic of the targetimage. In the implementation provided in FIG. 6, the characteristic ofthe target image may be the probability that the patient from whom thetarget image was captured may exhibit cancer recurrence. Alternativelyor in addition, the characteristic may be a degree of cancer recurrencethat the patient from whom the target image was captured may exhibit. At216, the characteristic may be output in accordance with the disclosureprovided herein.

FIG. 7 shows an example implementation of a specialized machine learningmodel, generated using a generalized machine model, for drug toxicity ortissue abnormality prediction. In drug development cycle, potentialcompounds go through rounds of safety studies in animals and thenhumans. For example, based on current practices, assessment of toxicityis conducted manually via pathology testing in animal tissues. Thenumber of animals in toxicity preclinical studies is quite limited andmay require testing of multiple doses of a new molecular entity.Techniques disclosed herein including a generalized machine learningmodel that was trained on various human tissues, animal tissues, or anyapplicable tissues, that is weighted based on learned the tumormorphology can be used as an initialization step to detect abnormalitiesin tissues (e.g., animal tissues) when making predictions forpreclinical toxicity studies. As shown in FIG. 7, at 702, a generalizedmachine learning model may be received at 702. The generalized machinelearning model may be adjusted to have drug toxicity or tissueabnormality prediction outputs at 704. The adjustment may be made byadjusting the weights in one or more layers of the generalized machinelearning model and/or the weights of the output layer of the generalizedmachine learning model, and/or modifying attributes of the outputs ofthe generalized machine learning model.

The adjusted machine learning model may be provided to training module300. The training module 300 may be configured to generate thespecialized machine learning model, at 710, by maintaining the one ormore layers (e.g., inner layers) of the generalized machine learningmodel and modifying one or more layers (e.g., the outer layers) of themachine learning model. The specialized machine learning model may betrained using a relatively small amount of data and may leverage thepreviously trained layers of the generalized machine learning model. Thespecialized machine learning model may be trained by providing pathologyimages and corresponding patient data for patients (e.g., human and/oranimal) that were provided a target drug. Additionally, known outcomesof the target drug's toxicity or resulting tissue abnormality may alsobe provided to the training module 300 to train the specialized machinelearning model.

At 710, the specialized machine learning model may be generated based onmodifying the generalized machine learning model received at 720 andtraining based on specialized training images from patients that areprovided the target drug, as well as their known toxicity or tissueabnormality from the drug. The specialized machine learning modelgenerated at 710 may be used to predict drug toxicity or tissueabnormality outcomes based on one more target images. FIG. 7 shows steps212, 214, and 216 of FIG. 2 and disclosure related to these steps is notrepeated here for brevity. At 212, a target image corresponding to atarget specimen may be received. At 214, the specialized machinelearning model generated at 710 may be applied to the target image todetermine a characteristic of the target image. In the implementationprovided in FIG. 7, the characteristic of the target image may be thepresence or absence, or degree of toxicity or tissue abnormality thatthe patient from whom the target image was captured may have to thetarget drug. At 216, the characteristic may be output in accordance withthe disclosure provided herein.

As shown in FIG. 8, device 800 may include a central processing unit(CPU) 820. CPU 820 may be any type of processor device including, forexample, any type of special purpose or a general-purpose microprocessordevice. As will be appreciated by persons skilled in the relevant art,CPU 820 also may be a single processor in a multi-core/multiprocessorsystem, such system operating alone, or in a cluster of computingdevices operating in a cluster or server farm. CPU 820 may be connectedto a data communication infrastructure 810, for example, a bus, messagequeue, network, or multi-core message-passing scheme.

Device 800 also may include a main memory 840, for example, randomaccess memory (RAM), and also may include a secondary memory 830.Secondary memory 830, e.g., a read-only memory (ROM), may be, forexample, a hard disk drive or a removable storage drive. Such aremovable storage drive may comprise, for example, a floppy disk drive,a magnetic tape drive, an optical disk drive, a flash memory, or thelike. The removable storage drive in this example reads from and/orwrites to a removable storage unit in a well-known manner. The removablestorage unit may comprise a floppy disk, magnetic tape, optical disk,etc., which is read by and written to by the removable storage drive. Aswill be appreciated by persons skilled in the relevant art, such aremovable storage unit generally includes a computer usable storagemedium having stored therein computer software and/or data.

In alternative implementations, secondary memory 830 may include othersimilar means for allowing computer programs or other instructions to beloaded into device 800. Examples of such means may include a programcartridge and cartridge interface (such as that found in video gamedevices), a removable memory chip (such as an EPROM, or PROM) andassociated socket, and other removable storage units and interfaces,which allow software and data to be transferred from a removable storageunit to device 800.

Device 800 also may include a communications interface (“COM”) 860.Communications interface 860 allows software and data to be transferredbetween device 800 and external devices. Communications interface 860may include a modem, a network interface (such as an Ethernet card), acommunications port, a PCMCIA slot and card, or the like. Software anddata transferred via communications interface 860 may be in the form ofsignals, which may be electronic, electromagnetic, optical, or othersignals capable of being received by communications interface 860. Thesesignals may be provided to communications interface 860 via acommunications path of device 800, which may be implemented using, forexample, wire or cable, fiber optics, a phone line, a cellular phonelink, an RF link or other communications channels.

The hardware elements, operating systems and programming languages ofsuch equipment are conventional in nature, and it is presumed that thoseskilled in the art are adequately familiar therewith. Device 800 alsomay include input and output ports 850 to connect with input and outputdevices such as keyboards, mice, touchscreens, monitors, displays, etc.Of course, the various server functions may be implemented in adistributed fashion on a number of similar platforms, to distribute theprocessing load. Alternatively, the servers may be implemented byappropriate programming of one computer hardware platform.

Throughout this disclosure, references to components or modulesgenerally refer to items that logically can be grouped together toperform a function or group of related functions. Like referencenumerals are generally intended to refer to the same or similarcomponents. Components and modules can be implemented in software,hardware, or a combination of software and hardware.

The tools, modules, and functions described above may be performed byone or more processors. “Storage” type media may include any or all ofthe tangible memory of the computers, processors or the like, orassociated modules thereof, such as various semiconductor memories, tapedrives, disk drives and the like, which may provide non-transitorystorage at any time for software programming.

Software may be communicated through the Internet, a cloud serviceprovider, or other telecommunication networks. For example,communications may enable loading software from one computer orprocessor into another. As used herein, unless restricted tonon-transitory, tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

The foregoing general description is exemplary and explanatory only, andnot restrictive of the disclosure. Other embodiments of the inventionwill be apparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. It isintended that the specification and examples be considered as exemplaryonly.

What is claimed is:
 1. A computer-implemented method for processing electronic images, the method comprising: receiving a generalized machine learning model generated by processing a plurality of first training images to predict at least one cancer characteristic; receiving a plurality of second training images, wherein the first training images and the second training images comprise images of tissue specimens and/or images algorithmically generated to replicate tissue specimens, a quantity of second training images being insufficient to generate a machine learning model that meets a threshold for outputting one or more characteristics of target images for a specialized task; receiving a plurality of target specialized attributes each related to a respective second training image of the plurality of second training images; generating a specialized machine learning model by modifying the generalized machine learning model based on the plurality of second training images and respective target specialized attributes, the specialized machine learning model meeting the threshold for outputting the one or more characteristics of target images for the specialized task, the specialized machine learning model being generated in accordance with a feature extraction scheme by generating additional layers to an existing set of layers of the generalized machine learning model and generating weights for the additional layers based on the plurality of second training images and the target specialized attributes; receiving a target image corresponding to a target specimen; applying the specialized machine learning model to the target image to determine at least one characteristic of the target image; and outputting the at least one characteristic of the target image.
 2. The computer-implemented method of claim 1, further comprising: determining a prediction of a specimen type of the target specimen based on the at least one characteristic of the target image; and outputting the prediction of the specimen type of the target specimen.
 3. The computer-implemented method of claim 1, wherein the plurality of target specialized attributes are one or more biomarkers present within each respective second training image.
 4. The computer-implemented method of claim 1, wherein the generalized machine learning model comprises a plurality of layers and modifying the generalized machine learning model further comprises modifying one or more outer layers of the generalized machine learning model.
 5. The computer-implemented method of claim 1, wherein modifying the generalized machine learning model further comprises removing an output layer of the generalized machine learning model.
 6. The computer-implemented method of claim 1, wherein the plurality of target specialized attributes are one or more indications of characteristic outputs selected from a disease presence, staging variable presence, drug response, toxicity, or cancer classification.
 7. The computer-implemented method of claim 1, wherein the plurality of target specialized attributes are based on at least one of drug response information, cancer recurrence prediction information, or toxicity assessment information.
 8. The computer-implemented method of claim 1, wherein each of second training images are generated based on a same category of pathology specimens.
 9. The computer-implemented method of claim 8, wherein a category of pathology specimens is selected from histology, cytology, immunohistochemistry, or a combination thereof.
 10. The computer-implemented method of claim 1, wherein modifying a generalized machine learning model further comprises adjusting the generalized machine learning model to have outputs based on the target specialized attributes.
 11. The computer-implemented method of claim 1, wherein the plurality of first training images comprise images corresponding to a plurality of cancer types.
 12. The computer-implemented method of claim 1, wherein the at least one cancer characteristic is one of a cancer diagnosis, a tumor characterization, or biomarker detection.
 13. A system for processing electronic images, the system comprising: at least one memory storing instructions; and at least one processor executing the instructions to perform operations comprising: receiving a generalized machine learning model generated by processing a plurality of first training images to predict at least one cancer characteristic; receiving a plurality of second training images, wherein the first training images and the second training images comprise images of tissue specimens and/or images algorithmically generated to replicate tissue specimens a quantity of second training images being insufficient to generate a machine learning model that meets a threshold for outputting one or more characteristics of target images for a specialized task; receiving a plurality of target specialized attributes each related to a respective second training image of the plurality of second training images; generating a specialized machine learning model by modifying the generalized machine learning model based on the plurality of second training images and the respective target specialized attributes, the specialized machine learning model meeting the threshold for outputting the one or more characteristics of target images for the specialized task, the specialized machine learning model being generated in accordance with a feature extraction scheme by generating additional layers to an existing set of layers of the generalized machine learning model and generating weights for the additional layers based on the plurality of second training images and the target specialized attributes; receiving a target image corresponding to a target specimen; applying the specialized machine learning model to the target image to determine at least one characteristic of the target image; and outputting the at least one characteristic of the target image.
 14. The system of claim 13, the operations further comprising: determining a prediction of a specimen type of the target specimen based on the at least one characteristic of the target image; and outputting the prediction of the specimen type of the target specimen.
 15. The system of claim 13, wherein the plurality of target specialized attributes are one or more biomarkers present within each respective second training image.
 16. The system of claim 13, wherein the generalized machine learning model comprises a plurality of layers and modifying the generalized machine learning model further comprises modifying one or more outer layers of the generalized machine learning model.
 17. The system of claim 13, wherein modifying the generalized machine learning model further comprises removing an output layer of the generalized machine learning model.
 18. A non-transitory computer-readable medium storing instructions that, when executed by processor, cause the processor to perform operations for processing electronic images, the operations comprising: receiving a generalized machine learning model generated by processing a plurality of first training images to predict at least one cancer characteristic; receiving a plurality of second training images, wherein the first training images and the second training images comprise images of tissue specimens and/or images algorithmically generated to replicate tissue specimens a quantity of second training images being insufficient to generate a machine learning model that meets a threshold for outputting one or more characteristics of target images for a specialized task; receiving a plurality of target specialized attributes each related to a respective second training image of the plurality of second training images; generating a specialized machine learning model by modifying the generalized machine learning model based on the plurality of second training images and the respective target specialized attributes, the specialized machine learning model meeting the threshold for outputting the one or more characteristics of target images for the specialized task, the specialized machine learning model being generated in accordance with a feature extraction scheme by generating additional layers to an existing set of layers of the generalized machine learning model and generating weights for the additional layers based on the plurality of second training images and the target specialized attributes; receiving a target image corresponding to a target specimen; applying the specialized machine learning model to the target image to determine at least one characteristic of the target image; and outputting the at least one characteristic of the target image.
 19. The non-transitory computer-readable medium of claim 18, the operations further comprising: determining a prediction of a specimen type of the target specimen based on the at least one characteristic of the target image; and outputting the prediction of the specimen type of the target specimen.
 20. The non-transitory computer-readable medium of claim 18, wherein the plurality of target specialized attributes are one or more biomarkers present within each respective second training image. 